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0 An apparatus for manipulating an object displayed on a display device. 



0 In an apparatus for manipulating an object dis- 
played on a display devices (3), 

a touch screen (11) which is sensitive at least to 
a position thereon at which a body touches, outputs 
touch screen information representing a motion of 
the body, 

a plurality of data files store object data for 
displaying the object in different states, 

a display information table (17) stores object 
information including an object type which specifies 
the shape and physical properties of the object, 
display position information which specifies a posi- 



tion where the object is displayed on the display 
device, file information which specifies the size and 
location of a part of the object data stored in one 
said plurality of data files, and a file name which 
specifies one of said plurality of data files, and 

a display controller (52) recognizes a manipula- 
tion to be conducted on the object based on the 
touch screen information from the touch screen and 
on the object information included in the display 
information table, and displays the object on the 
display device in accordance with the recognition. 
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The present invention relates to an apparatus 
for manipulating an object displayed on a display 

device. 

As use of computer systems for data process- 
ing has become widespread in recent years, more 
and more users are being required to input data to 
and converse with data processors such as work 
stations and personal computers. A vast range of 
application programs are available for recent data 
processors and even a complicated application can 
be processed by using such application programs 
in combination. However, there is a problem that 
such data processors are very difficult to handle, 
especially to manipulate an object displayed on a 
display device, for those who have little knowledge 
of computers. 

Therefore, an apparatus for manipulating an 
object displayed on a display device, which is easy 
to use even for a person who has no special 
knowledge of computers, is in great demand. 

Fig. 1 illustrates a computer system with a 
conventional user interface. 

A computer system with a conventional user 
interface consists mainly of a central processing 
unit (CPU) 4, a main memory 5, a keyboard/mouse 
2, a frame memory 60 and a hard disk interface 71, 
which are interconnected via a system bus inter- 
face, and also a hard disk 7 and a display unit 3, 
which are connected to system bus interface via 
the hard disk interface and the frame memory 6, 
respectively. The main memory 5 stores a system 
control program and application programs which 
handle graphics processing, and provides a work 
area for use by the programs. The CPU 4 performs 
display operations under control of the programs. 
The hard disk 7 stores a data file for graphics to be 
displayed on the display unit 3. The frame memory 
6 stores a frame of picture (or object) data to be 
displayed on the display unit 3. 

To manipulate an object displayed on a display 
unit 3 in the above system, an operator is required 
to input a command for manipulating the object by 
using a keyboard/mouse 2 or to select an icon (a 
symbolic representation of a computer function) 
displayed on a display unit 3 by using the 
keyboard/mouse 2 in order to command a desired 
function. However, it is troublesome and annoying 
to use a keyboard/mouse and icons and a person 
with less knowledge of computers tends to be 
allergic to even touching a keyboard/mouse. 

Therefore, it is a great problem that such data 
processors are very difficult to handle for those 
who have less knowledge of computers. 

It is therefore desirable to provide an apparatus 
which can easily manipulate an object displayed on 
a display unit. 

It is also desirable to provide a user interface 
with which a user can easily manipulate an object 



displayed on a display unit. 

The present invention provides a touch-sen- 
sitive panel (e.g. touch screen), means storing a 
plurality of data files, display information storage 

5 means and display control means. 

In the apparatus of the present invention, the 
touch panel, which may for example be provided 
on a display surface of the display device and is 
sensitive at least to a position thereon at which a 

70 body (e.g. finger) touches, outputs touch panel 
information representing motion of the body. The 
plurality of data files store object data for display- 
ing the object in different states. The display in- 
formation storage means stores object information 

75 including at least an object type which specifies the 
shape and physical properties of the object, display 
position information which specifies a position 
where the object is displayed on the display de- 
vice, file information which specifies the size and 

20 location of a part of the object data stored in one of 
said plurality of data files, and a file name which 
specifies one of said plurality of data files. The 
display control means recognizes a manipulation to 
be conducted on the object based on the touch 

25 panel information from the touch panel and on the 
object information included in the display informa- 
tion storage means, and displays the object on the 
display device in accordance with the recognition. 
Reference is made, by way of example, to the 

30 accompanying drawings, in which:- 

Fig. 1 illustrates a computer system with a con- 
ventional user interface; 

Fig. 2 is a configuration diagram of a touch- 
screen-equipped workstation, to which the 
35 present invention may be applied; 

Fig. 3 is a schematic diagram illustrating the 

principle of the present invention; 
Fig. 4(a) shows a display information table; 
Fig. 4(b) shows touch screen information; 
40 Fig. 5 is a flowchart illustrating a pick manipula- 
tion; 

Fig. 6 is a diagram illustrating a pick manipula- 
tion; 

Fig. 7 is a diagram illustrating a scroll manlpula- 
45 tion; 

Fig. 8 is a diagram illustrating a push manipula- 
tion; 

Fig. 9 is a diagram illustrating a flip manipula- 
tion; 

50 Fig. 10 is a diagram illustrating a roll manipula- 
tion; and 

Fig. 11 is a diagram illustrating a distort-restore 
manipulation. 

Throughout the above-mentioned drawings, 
65 identical reference numerals are used to designate 
the same or similar component parts. 

Fig. 2 is a configuration diagram of a touch 
screen-equipped workstation for implementing the 
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present invention. 

In addition to the conventional system shown in 
Fig. 1, the system includes an input-output 
(abbreviated to I/O) port 8, a touch screen control- 
ler 15 and a touch screen unit 1 with a touch s 
screen 11. The touch screen controller 15, con- 
nected to the input-output port 8 through an RS- 
232C interface, controls the touch screen unit 1. 
The touch screen unit 1, which is sensitive to a 
position or positions p<-Y co-ordinates) where It Is w 
touched, and preferably also to a pressure applied 
to it, acts as a user interface that allows a user to 
send signals to the CPU by touching an area 
thereon with a body, such as a finger or a pencil. 

Fig. 3 is a schematic diagram illustrating the 75 
principle of the present invention. 

For easy understanding of the principle, the 
Input-output port 8, touch screen controller 15 and 
touch screen unit 1 shown in Fig. 2 are repre- 
sented by the touch screen unit 1; and the frame 20 
memory 6 and display unit 3 are represented by 
the display unit 3. A the system controller 50, 
touch discriminator 51 display controller 52 and 
display information table IT, which are stored in 
the main memory 50, control display operations 25 
featured by the present invention. 

Fig. 4(a) shows a display information table. Fig. 
4(b) shows touch screen information. 

A display information table 1T, which is pro- 
vided in the main memory 5, corresponding to 30 
objects, Includes an object type, display position 
information, file information, normal-display file 
name and special-state file name. The object type 
defines the type including the shape, properties, 
circumstances, etc., of the object. The display posi- 35 
tion Information defines the size of the object 
(width, height), and the position (top-left coordi- 
nates X,Y) and the angle at which the object is 
displayed on the display unit 1 . The file information 
defines the size (width, height) the object data to 40 
be displayed occupies in the display data file, and 
also the position (top-left coordinates X,Y) where 
the object data to be displayed is stored in the 
display data file. The normal-display file name 
specifies a display data file where object data for 45 
displaying a normal state of the object is stored. 
The special-state file name specifies a display data 
file where object data for displaying a special state 
(e.g., turn-over indication of display color, used for 
displaying intermediate process of manipulating the so 
object) of the object is stored. 

Touch-screen information 21, which is sent from 
the touch screen unit 1 , includes a touch position 
(X-Y coordinates) where the touch screen 11 Is 
touched and a pressure applied thereon. 55 

Following embodiments of the present inven- 
tion correspond, item for item, to the Claims recited 
later. 



(1) The touch discriminator 51, based on the 
touch screen information 21 from the touch 
screen unit 1, discriminates the type of a touch 
an operator's finger has on the touch screen 1 1 , 
that is, a touch type including, i.e., a 
"continuous touch start" and "continuous touch 
end" explained later. The touch discriminator 51 
sends to the system controller 50, the result of 
the discrimination as a touch report 3R, which 
includes a touch type and touch coordinates. 

Based on the touch report 3R from the 
touch discriminator 51 and the display informa- 
tion table 1T, the system controller 50 deter- 
mines the type of a manipulation conducted by 
an operator and, according to the determination, 
updates the display information table IT. Then, 
the system controller 50 sends to the display 
controller 52, a display update request 4Q along 
with "display update data" which includes con- 
tents of the display information table IT updated 
(including display position information, file in- 
formation arid file name). 

On receipt of the display update request 40 
from the system controller 50, the display con- 
troller 52 reads a display data file (including 
object data) specified by the file name from the 
hard disk 7 and stores the data into the main 
memory 5. The display controller 52 then up- 
dates the object data in accordance with the 
display update data from the system controller 
50 and loads the thus-updated object data Into 
the frame memory 6 to display the object, as 
manipulated by the operator on the touch 
screen unit 1 . 

Thus, the present invention determines a 
manipulation to be conducted on the object dis- 
played, based on the touch screen information 
21 which results from an operator's touching the 
touch screen 1 1 and the display information 
table IT which defines the object's shape, phys- 
ical properties, display position, etc. It then dis- 
plays the object according to the manipulation 
determined, as intended by the operator. 

(2) Pick manipulation (see Figs. 5 and 6.) 

A pick manipulation is conducted in such a 
way as an object is picked up at a position on 
the display surface of the display unit 3 and 
placed at another position. 

Fig. 5 is a flowchart illustrating a pick ma- 
nipulation. Fig. 6 is a diagram illustrating a pick 
manipulation. 

A pick manipulation is carried out according 
to the following steps (S1-S8) in Fig. 5: 

(51) The system controller 50 receives a 
touch report 3R from the touch discriminator 
51. 

(52) The system controller 50 checks the 
touch report 3R to see whether the object- 
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finger relation is a pick manipulation as 
shown in Fig. 6(a), based on the touch report 
3R and contents of the display information 
table 1T shown in Fig. 6(c). When the relation 
is not a pick manipulation, the system con- 5 
troller 50 checks the touch report 3R for other 
manipulation. 

(53) When the relation is a pick manipulation, 
the system controller 50 sends a display up- 
date request 4Q Including "display update io 
data", commanding that the special-state file 
(turn-over indication) be displayed at the po- 
sition specified by the display information 
table 1T. 

(54) The system controller 50 receives a 75 
touch report 3R. 

(55) The system controller 50 determines 
whether the touch report 3R includes a 
"continuous touch end", which occurs when 

the finger-object relation is as in Fig. 6(b). 20 
When a "continuous touch end" is reported, 
the operation goes to step (S8). 

(56) Otherwise, the system controller 50 up- 
dates the display position information 
"coordinates (X, Y)" of the display informa- 25 
tion table 1T so that the object is positioned 
between the two fingers. 

(57) The system controller 50 sends display 
update request 4Q to the display controller 

52, commanding that the special-state file be 30 
displayed according to the display informa- 
tion table 1T updated, and returns to step 
(S4). 

(58) When "continuous touch end" is re- 
ported by a touch report 3R, the system 35 
controller 50 sends a display update request 

4Q to the display controller 52, commanding 
that the normal-display file be displayed at 
the position specified in the display informa- 
tion table 1T. 40 

Following manipulations are carried out in 
the same way as described in the above 
flowchart of the pick manipulation. 
(3) Scroll manipulation (see Fig. 7.) 

A scroll manipulation Is conducted in such a 45 
way as an object extending outside of the dis- 
play surface of the display unit 3 is moved into 
and out of the display surface. 

Fig. 7 is a diagram illustrating a scroll ma- 
nipulation. 50 

On determining that the finger moves while 
touching the touch screen 11 based on the 
touch screen information 21 from the touch 
screen unit 1 , the discriminator 51 sends to the 
system controller 50, a touch report 3R includ- 55 
ing "continuous touch start" for the touch type 
and also "coordinates (800, 800)" for the touch 
position. As another touch screen information 21 



comes in, the discriminator 51 sends a touch 
report 3R Including "continuous touch In 
progress" and coordinates (780, 800). When the 
touch screen information 21 is not sent for more 
than 100 milliseconds, for example, the discrimi- 
nator 51 sends a touch report 3R including 
"continuous touch end" and coordinates (700, 
800) to the system controller 50. 

when a "continuous touch start" is reported 
and the "object type" is defined as "out-screen" 
in the display information table 1T, the system 
controller 50 recognizes the object as a large 
one extending beyond the display screen. Then, 
the system controller 50 determines the speed 
at which the finger has moved from right to left, 
for example, based on a change in the X-coordi- 
nate in the touch report 3R. 

Depending on whether the finger has moved 
at a speed of more (high-speed) or less (normal- 
speed) than for example 20 dots (pixels) e.g. 
since the last check, the display screen is scrol- 
led first at an interval of 100 or 500 milliseconds, 
respectively. Then, the interval, at which the 
display update request 40 is sent to the display 
controller 52, is increased by a factor 1.5 at 
each touch report 3R and, when the interval 
reaches 2 seconds, the scrolling is stopped. 

Practically, the screen Is so controlled that it 
starts scrolling at an above-mentioned speed 
after a finger has moved a distance of 4 dots or 
more. That is, on recognizing that the finger has 
moved for that distance, the system controller 
50 updates the file information "display position 
X" of the display information table IT so that 
the object is displayed to the left by 10 dots, for 
example. Then, It sends to the display controller 
52, a display update request including display 
position information, file information and normal 
display file name from the display Information 
table IT updated. 

The display controller 52 reads from the 
hard disk a display file specified by the normal 
display file name and loads it in the main mem- 
ory 5. The display controller 52 then transfers 
only the part of the display file specified by the 
file Information "display position X" of the dis- 
play information table 1T, from the main mem- 
ory 5 to the appropriate location of the frame 
memory 6. 

In the same way, the system controller 50 
sends a display update request 40 to the dis- 
play controller 52 every time it receives a touch 
report 3R . 

When another "continuous touch" is report- 
ed before the scroll currently in progress comes 
to a stop, a new scroll can start from this point 
and at the first speed described above. 
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(4) Scroll-Stop manipulation (see Fig. 7.) 

Fig. 7 is a diagrann illustrating a scroll nna- 

nipulation. 

When a touch position given by a touch 
report 3R is the sanne as or approximately 5 6 
dots apart from the position of the scrolling 
currently in progress, the system controller 50 
doubles the frequency with which display up- 
date requests 4Q are sent to the display control- 
ler 52, in order to put an end to the scrolling. w 

(5) Push manipulation (see Fig. 8.) 

A push manipulation is conducted In such a 
way as an object is pushed on the display 
surface of the display unit 3. 

Fig. 8 is a diagram illustrating a push ma- 75 
nipulation. 

The system controller 50 determines the 
type of a manipulation, based on the touch 
report 3R and contents of the display informa- 
tion table 1T shown in Fig. 8(c). When the 20 
manipulation is a push manipulation as shown in 
Fig. 8(a), the system controller 50 sends to the 
display controller 52, a display update request 
4Q including display position information, file 
Information and normal display file name so that 25 
the object is displayed close to the finger posi- 
tion reported by the touch report 3R. The above 
display operation is repeated until a "continuous 
touch end" is reported by a touch report 3R. 

(6) Push-while-rotate manipulation (see Fig. 8.) 30 
A push-while-rotate manipulation is conduct- 
ed in such a way as an object is pushed at a 
position off its center (or the center of gravity) 

and it moves rotating on the display surface of 

the display unit 3. 35 

Fig. 8 is a diagram illustrating a push ma- 
nipulation. 

The system controller 50 determines the 
type of a manipulation, based on the touch 
report 3R and contents of the display informa- 40 
tion table 1T shown in Fig. 8(c). When the 
manipulation is a push-while-rotate manipulation 
as shown in Fig. 8(b), the system controller 50 
sends to the display controller 52, display up- 
date requests 4Q with the angle of rotation in- 45 
creasing by 2 degrees, i.e., while increasing the 
angle in the display information table 1T shown 
In Fig. 8(c). 

The display controller 52 reads the display 
file from the hard disk and loads the data in the 50 
main memory 5, rotates the object by the angle 
and with the left-top coordinates (X, Y) as a 
rotational center, as specified by the display 
update request 4Q, and transfers the data with 
the object rotated, from the main memory 5 to 55 
the frame memory 6. 

(7) Flip manipulation (see Fig. 9.) 

A flip manipulation is conducted in such a 



way as a finger flips an object or touches the 
object from a remote position at a high speed 
on the display surface of the display unit 3. 

Fig. 9 is a diagram illustrating a flip manipu- 
lation. 

When a touch report 3R is input from the 
touch discriminator 51 , the system controller 50 
descriminates the type of a manipulation based 
on the touch report 3R and contents of the 
display information table 1T shown in Fig. 9 (c). 
When the manipulation is a flip manipulation as 
shown in Fig. 9 (a), the system controller 50 
obtains a finger speed based on the touch re- 
port 3R and also an object speed (i.e., the 
interval at which display update requests 4Q are 
sent to the display controller 52), in the same 
way as described in item (3). The system con- 
troller 50 sends display update requests 4Q to 
the display controller 52, while updating the 
display position information left-top coordinates 
(X, Y) of the display information table 1T so that 
the object moves in the direction the finger 
moves. The system controller 50 stops moving 
the object when the above-mentioned interval 
reaches 2 seconds. 

(8) Flip-under-gravity manipulation (see Fig. 9.) 
A flip-under-gravity manipulation is conduct- 
ed in such a way as an object which is sub- 
jected to a gravity is flipped by a finger on the 
display surface of the display unit 3. 

Fig. 9 is a diagram illustrating a flip manipu- 
lation. 

When the finger manipulation is a flip as in 
the above item (8) and the display information 
table 1T defines the object type as "gravity" 
meaning that the object is subjected to gravity, 
for example, the object moves under the com- 
bined influences of inertia and simulated gravity, 
i.e. "falls" as shown in Fig. 9(b). Therefore, the 
system controller 50 sends display update re- 
quests 4Q to the display controller 52, while 
updating the display position information left-top 
coordinates (X, Y) by adding a value to the Y- 
coordinate of of the display information table 1T. 
The value is represented by 2 to the Nth power 
(N: the number of display-update requests 4Q 
sent). In this case, too, the system controller 50 
stops moving the object when the above-men- 
tioned interval reaches 2 seconds. The resulting 
trajectory may be a parabola. 

(9) Roll manipulation (see Fig. 10.) 

A roll manipulation is conducted in such a 
way as a reliable object is rolled by a finger on 
the display surface of the display unit 3. 

Fig. 10 is a diagram illustrating a roll ma- 
nipulation. 

When a touch report 3R is input from the 
touch discriminator 51 and the display informa- 
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tion table 1T defines the object type as 
"tollable" meaning that the object is constructed 
such that it rolls when flipped like a globe or a 
cylinder, as shown in Fig. 10(a), the system 
controller 50 sends display update requests 4Q 
to the display controller 52, while updating the 
display position information left-top coordinates 
(X, Y) of the display information table IT so that 
the object moves a distance 10 per cent behind 
the distance and In the direction the finger 
moves. 

(10) Distort-restore manipulation (see Fig. 11.) 

A distort-restore manipulation is conducted 
in such a way as an "elastic "object is pressed 
by a finger on the display surface of the display 
unit 3, thereby deforming the displayed object. 

Fig. 11 is a diagram illustrating a distort- 
restore manipulation. 

When a touch report 3R is input from the 
touch discriminator 51 and the display informa- 
tion table IT defines the object type as "elastic" 
meaning that the object can be distorted and 
restored according to a pressure applied there- 
on by a finger, as shown in Fig. 11(a), the 
system controller 50 calculates an amount of 
distortion of the object based on the pressure 
reported by the touch report 3R. It stores in the 
display information table IT, a special-state file 
name specifying one of special-state files (for 
displaying a distorted state of the object in turn- 
over indication) corresponding to the amount of 
distortion calculated. Then, the system controller 
50 sends a display update request 4Q to the 
display controller 52, commanding that the 
special-state file be displayed at the current 
display position. When the above operation is 
repeated as necessary and a "continuous touch 
end" is reported by a touch report 3R, the 
system controller 50 sends a display update 
request 4Q (with a normal display file name 
specified) to the display controller 52, com- 
manding that a normal display file (normal in- 
dication) be displayed at the current display 
position. A plurality of special-state files are pro- 
vided in the hard disk 7, corresponding to the 
amount of distortion of the object, which results 
from a pressure applied on the touch screen 1 1 . 
As is apparent by the above description, the 
present invention regards a display screen as a 
virtual space. It defines conditions and physical 
properties of an object (e.g., weight, hardness, fric- 
tional resistance, center of gravity) in the display 
information table IT. It also receives touch screen 
Information 21 Indicating a finger-touched position 
and pressure is input from a touch screen unit 1. 
Based on the touch screen information 21 and the 
display information table IT, the present invention 
determines a manipulation to be conducted on the 



object displayed, e.g. scrolling, picking (up), push- 
ing, rolling, distorting the object on the display 
surface of the display unit 3. Thus, the present 
invention allows a user to manipulate an object 

5 displayed on a display device quite easily, even 
when the user has little knowledge of computers. 

The above description refers to a touch screen 
on the face of a display device, and this is the 
most preferable arrangement from the viewpoint of 

70 ease of use. However, the present invention is not 
limited to a touch screen, since it can also be 
applied using a touch panel quite separate from the 
display device, e.g. a graphics tablet. In this case, 
a finger or other body touching the panel may be 

75 represented by a symbol on the display device, so 
as to allow manipulation of objects on the display 
simply by observing the display. 

Claims 

20 

1. An apparatus for manipulating an object dis- 
played on a display device, comprising:- 

a touch panel, representing an area of a 
display surface of the display device and sen- 
25 sitive at least to a position on the panel at 

which a body touches, for outputting touch 
panel information representing motion of the 
body; 

means for storing a plurality of data files 
30 which store object data for displaying the ob- 

ject in different states; 

display information storage means for stor- 
ing object information including at least:- 

an object type which specifies the shape 
35 and physical properties of the object, 

display position information which speci- 
fies a position where the object is displayed on 
the display device, 

file information which specifies the size 
40 and location of a part of the object data stored 

in one of said plurality of data files, and 

a file name which specifies one of said 
plurality of data files; and 

display control means, responsive to the 
45 touch panel information and to the object in- 

formation included in said display information 
storage means, for recognizing a manipulation 
to be conducted on the object and for display- 
ing the object on the display device in accor- 
50 dance with the recognition. 

2, An apparatus according to claim 1 , wherein the 
touch panel is a touch screen which is pro- 
vided on the display surface of the display 

65 device. 

3- An apparatus according to claim 2, wherein in 
response to touch panel information indicating 
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that two bodies touch both sides of the object, 
move and stop with a distance in between, 
said display control means is operable to rec- 
ognize a pick manipulation and to display the 
object on the display device so that the object 
moves on the display surface of the display 
device from where the two bodies touch both 
sides of the object to where the two bodies 
stop with a distance in between. 

4. An apparatus according to claim 2 or 3, 
wherein in response to the touch panel in- 
formation indicating that a body touches the 
object and moves on said touch panel while 
keeping in contact with the object, and in re- 
sponse to object information specifying the 
object type as "out-of-screen" in said display 
information storage means, which means that 
the object is a large one extending beyond the 
display screen, said display control means rec- 
ognizes a scroll manipulation and displays the 
object on the display device so that the object 
scrolls on the display surface of the display 
device. 

5. An apparatus according to claim 4, wherein in 
response to touch panel information from said 
touch panel Indicating that the moving body 
stops, said display control means recognizes a 
scroll-stop manipulation and displays the ob- 
ject on the display device so that the scrolling 
object stops on the display surface of the 
display device. 

6. An apparatus according to any of claims 2 to 

5, wherein in response to touch panel informa- 
tion indicating that a body touches the object 
at its center or center of gravity, moves and 
stops on said touch panel while keeping in 
contact with the object, said display control 
means recognizes a push manipulation and 
displays the object on the display device so 
that the object moves on the display surface 
from where the body touches the object to 
where the body stops. 

7. An apparatus according to any of claims 2 to 

6, wherein in response to touch panel informa- 
tion indicating that a body touches the object 
at a position off the center or the center of 
gravity thereof, moves and stops on said touch 
panel while keeping in contact with the object, 
said display control means recognizes a push- 
while-rotate manipulation and displays the ob- 
ject on the display device so that the object 
moves while rotating on the display surface 
from where the body touches the object to 
where the body stops. 



8. An apparatus according to any of claims 2 to 
7, wherein in response to touch panel informa- 
tion from said touch panel indicating that a 
body touches the object from a position apart 

6 therefrom at a speed higher than a predeter- 

mined speed, said display control means rec- 
ognizes a flip manipulation and displays the 
object on the display device so that the object 
moves a distance proportional to the speed 

w with which the body touches the object and in 

the direction toward which the body touches 
the object. 

9. An apparatus according to any of claims 2 to 
75 8, wherein in response to touch panel informa- 
tion from said touch panel indicating that a 
body touches the object from a position apart 
therefrom at a speed higher than a predeter- 
mined speed, and in response to object in- 

20 formation specifying the object type as 

"gravity" in said display information storage 
means, which means that the object is subject 
to gravity, said display control means recog- 
nizes a flip-under-gravity manipulation band 

25 displays the object on the display device so 

that the object moves a distance proportional 
to the speed with which the body touches the 
object and along a trajectory like that which the 
object would describe if it were a physical 

30 object moving under the force of gravity. 

10. An apparatus according to any of claims 2 to 

9, wherein in response to touch panel informa- 
tion indicating that a body touches the object, 

35 moves and stops on said touch panel while 

keeping in contact with the object, and in re- 
sponse to object information specifying the 
object type as "reliable" in said display in- 
formation storage means, said display control 

40 means recognizes a roll manipulation and dis- 

plays the object on the display device so that 
the object moves with positional relations be- 
tween the object and the body varying from 
those at the beginning as the body moves. 

45 

11. An apparatus according to any of claims 2 to 

10, wherein said touch panel is sensitive to an 
amount of pressure applied thereon, and 
wherein in response to touch panel information 

50 indicating that a body touches the object with 

an amount of pressure, and in response to 
object information specifying the object type 
as "elastic" In said display information storage 
means, said display control means recognizes 

55 a distort- restore manipulation and displays the 

object on the display device so that the object 
varies in a degree of distortion and/or restora- 
tion according to the amount of pressure ap- 
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